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@ Source coding (Data compression) 
92 Information content, entropy 
93 Typicality and the source coding theorem 
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®7 Capacity of a noisy channel 
78 The noisy-channel coding theorem 
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@Player chooses door 1 


(Host opens door 3, revealing nothing, as promised 


an 


этлек? 


A should stick 

B should switch 

C makes no difference 
Z don't know 


Solving Three Doors by Probability Theory 


@Bayes's theorem 
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Solving Three Doors by Probability Theory 


@Bayes's theorem 
postenon “KELIHOOD PRIOR 
P(H|Data) = P(Data|H) P(H) 
P(Data) 


HYPOTHESIS PRIOR LIKELIHOOD ^ POSTERIOR 
H P(H) P(D=3|H) P(H|D=3) 
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P(D=3) = ኔ P(D=3|H)P(H) 
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Information measures 


H(X,Y) 


H(X) 


H(Y) 


H(X|Y) 


I(X;Y) 


H(Y|X) 


Communication over noisy channels 


@Noisy channels 82% 


inference 
“information measures 


Inference for channels 


Channel Q defines conditional probabilities 


P(y| x) 


If we choose an input distribution P(x), 
we have a joint distribution 

P(x,y) = P(x) P(y|x) 
with which we can do inference. 
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Information measures for noisy channels 


Channel Q defines conditional probabilities 
P(y |x) = 
If we choose an input distribution Р(х), 
we have a joint distribution 
Р(х,у) = P(x) P(y| x) 


for which we can compute the mutual information 


I(X;Y) = H(X)-H(X|Y) 
= H(Y)-H(Y|X) 
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Information measures for noisy channels 


Channel Q defines conditional probabilities 
P(y |x) = 
If we choose an input distribution Р(х), 
we have a joint distribution 
Р(х,у) = P(x) P(y| x) 


for which we can compute the mutual information 


I(X;Y) = H(X)-H(X|Y) 
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H(X,Y) 
H(X) 
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Mutual information for the BSC 
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በመያ 
Consider f = 0.1. 


1 1 


Assume input distribution Px = (po.pi) = {0.9,0.1}. 


Find 
ы I(X;Y) = H(Y)-H(Y|X) ү) 
= H2(0.18) — H2(0.1) 
0.68 — 0.47 > 


0.21 bits. 
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Complete the curve 


The Capacity of a channel 


is the maximum, over all input distributions P(z), 
of the mutual information: 
C = maxI(X;Y) 
Px 
The distribution P% that achieves the maximum is 
called the optimal input distribution. 


The Capacity of a channel 


is the maximum, over all input distributions P(z), 
of the mutual information: 


С = max I(X;Y) 
Px 


The distribution P% that achieves the maximum is 
called the optimal input distribution. 


@Shannon's noisy-channel coding theorem: 


Reliable (virtually error-free) communication 
is possible at rates up to C 


Ternary confusion channel 
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Assume input distribution Px = {4 


e Ify= M, what's г? 


ee What is the mutual information I(X;Y)? 
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Noisy typewriter 


A A 
B B 
ዕጫ ር 
5 D 
E E 
F F Pl(y=F|z=6) = 1/3; 
G G P(y=G|z=6G) = 1/3; 
H 5 Р(у-Н|г-6) = 1/3; 


‘What is the optimal input distribution? 


What is the capacity? 


Recommended homework 


(Reading: Chapters 1-6; Advance reading: Chapters 8, 9, 10 
@Arithmetic coding: 6.17 (p125) 6.3 (p118) 6.7 (p123) 


(Data compression recap: 
95.26 (p103) 
‘Invent a question about data compression. 


25969 handout 2 on website for more recommended exercises 


Huffman programs huffman.p, huffman.py are on website 
also a ‘bent coin file 0010000... as a compression benchmark 


